1. Technical Field
The present invention relates to data transmission, to fiber optic data transmission, and more particularly to the use of optical burst switching to provide an optical networking system.
2. Abbreviations
Certain terms are hereby defined as used in this specification.
OBS: optical burst switching
APS: automatic protection switching
TXR: transmitter
RXR: receiver
TXN: transmission
Tell-and-go, or TAG: a scheme for allocating paths through a networking system in which a node does not check with any external authority to gain permission before transmitting.
Reference Network: the inventive work disclosed in patent application Ser. No. 10/118,084.
MESH: those portions of the protocol and structure disclosed in the present application which have to do with adapting and extending the Reference Network to function in a network topology which has more links than a ring topology.
Outer network, outer protocol: those portions of the structure and functionality disclosed in the Reference Network patent application which have to do with protection and survivability, plus the MESH structure and functionality disclosed here.
Inner network, inner protocol: those portions of the structure and functionality of the Reference network, along with inventive work disclosed here, exclusive of protection, survivability, and mesh structure and functionality. The inner network comprises five “layers.” Starting with the lowest layer, the functionality and structure of the five layers builds each upon the last.FIXED: the first, lowest layer of the inner network, disclosed in the Reference network.TUNABLE: the second layer of the inner network.TANDEM: the third layer of the inner network.URGENCY: the fourth layer of the inner network.RESERVATION: the fifth layer of the inner network.Low-power network: any network or protocol which includes the TUNABLE or higher layers of the inner network.active ring: for a given channel, the particular ring upon which the network is advertising available paths. Only one active ring is allowed per channel at any given moment. The active ring is the only ring upon which a path may be set up. The active ring may be changed on the fly by the node with the token-in-hand when certain criteria are met.Ring Identifier, or RI: a field added to the token, identifying which ring is currently the active ring. (The term “RI” may be used as shorthand for an active ring.)primary ring: a distinguished Hamiltonian cycle in a given network of nodes. (An Hamiltonian cycle is a closed path, with no node visited twice, and every node visited once. The fact that each link in the cycle is traversed once and only once is implicit. More than one Hamiltonian cycle may exist in the same network. The primary ring is simply one of these.) The primary data cycle for a channel.chord: a link in the network which does not lie on the primary ring.contraction: a change in the data cycle (“active ring”) to bypass certain nodes and links by routing across a chord.bypass: the chord used in a contraction, or the contraction itself.expansion: a change in the data cycle to include certain nodes and links that were formerly excluded by contraction.snapback: expansion.available rings: all the rings usable for a particular wavelength, including the primary ring and a pre-computed, globally known subset of rings made possible by the available chords.grayed-out: a term referring to the status of COMMPATH records which are on the primary ring but not on the active ring.ONEROTATION: the time or distance involved in one complete rotation of a token around the networkingONEROTATIONBITS: the number of bits that can be transmitted in ONEROTATION time
3. Description of Related Art
Customers can access a network from a variety of locations. For example, a customer might enter a network from the Internet or a public switched telephone network (PSTN). That customer might request a transfer of data from a variety of sources. These sources might include a storage area network (SAN), a wide area network (WAN) or a local area network (LAN) that is also connected to the network. A number of network architectures have been developed to assist the transfer of data from such a source to such a sink.
A typical network architecture is shown in FIG. 1 (0100) and has one or more network nodes (0101, 0102, 0103, 0104) having primary (0111, 0112, 0113, 0114) and/or secondary (protective) (0121, 0122, 0123, 0124) communication data links. For purposes of the present application, the term communication channel shall be designated Ci where i takes on any value from 0 to (W+1). Each link, for instance link 0121, can include a number of communication channels (C0 to CW+1). Likewise, a communication path can be established from one node to another over these links via a given channel Ci.
While the present invention is particularly amenable to applications in situations where ring networks as illustrated in FIG. 1 (0100) are implemented with three or more nodes, it is also applicable to situations where point-to-point network communications are implemented. Additionally, while the focus of many preferred embodiments of the present invention concentrates on optical communications (including Synchronous Optical Networks (SONET) and associated topologies), the present invention is not limited to the use of optical fiber as the communication medium.
4. Unidirectional Path-Switched Ring (0200, 0300)
Referencing FIG. 2, a topology for a SONET ring called a unidirectional path-switched ring is illustrated (0200). This ring uses two optical fibers (0201, 0202) and is configured such that each fiber channel sends communications traffic in one direction such that the direction of communications is opposite between the two fibers (0201, 0202).
The network elements (0231, 0232, 0233, 0234) are generalized communication sources/sinks in this topology, and should be considered as such in the context of the present invention. The network interface modules (0211, 0212, 0213, 0214) generally represent a number of multiply redundant communication ports that permit multiplexing of received data from one of several ports in the event of a node or path failure. Multiplexing functions (0221, 0222, 0223, 0224) are generally performed electrically but may also be optically actuated. Note especially that SONET technology is a hop-by-hop system, where optical transmission is terminated at each node for electronic processing of every signal, and that the control and signaling information is in-band, i.e., the data frames themselves contain “overhead” information which is used for operations, administration, management & protection functions (OAM&P).
While the topology illustrated in FIG. 2 (0200) provides some insight into how data is transmitted from node to node during normal operation, the topology under link failure conditions in which a fallback/recovery mode of operation is activated may be seen in FIG. 3 (0300). Here the link failure has been introduced between two network interface units (0312, 0313). This failure prompts the network system management components to reconfigure the multiplexer switches (0321, 0322, 0323, 0324) to avoid the failing link if possible. Redirection of the receive data switch (0323) permits the network to recover from this condition and still permit transmit and receive connectivity to be maintained between all nodes in the network. This reconfiguration process is not instantaneous, however, and the network elements (0331, 0332, 0333, 0334) will experience some degree of latency during the crossover of the receive data switch (0323). Furthermore, while this topology provides for secondary (protective) communications backup capacity, it makes no provision for the idle bandwidth in these communication channels to be actively used by the network. Furthermore, SONET topology assumes that a given link is either “lit” or “dark”, meaning there is no provision for graceful degradation of a communication channel link in this paradigm. Neither is there a provision for the “part-time” usage of a communication channel in this paradigm.
5. Bi-Directional Line-Switched Self-Healing Ring (0400)
Another SONET ring topology that is widely used in the prior art is termed Bi-Directional Line-Switched Self-Healing Ring (BLSR) and is illustrated in the four-fiber ring of FIG. 4 (0400). In this configuration, some of the fiber is acting as stand-by (protection) (0411), in the event that the working fiber (0412) (or a node) fails. The protection copy (0411) becomes the working copy (0412) and traffic is diverted around the problem should a failure occur using the add/drop multiplexers (ADMs) (0401, 0402, 0403, 0404). As with all SONET approaches, this approach makes for a very robust system and provides high reliability, albeit at the increased cost of the addition of redundant fiber links (0411), and at the significant cost of electronic equipment, electrical power supply, footprint (space requirements) and air conditioning to process all data electronically at each node. Indeed, these drawbacks obtain with all current hop-by-hop, full optical-to-electronic-to-optical signal conversion approaches.
6. Fault-Tolerant Switching Methods (0500)
Since optical fiber has a very large bandwidth and associated information carrying capacity, along with the capability of supporting a wide variety of simultaneous logical data connections, the loss of the fiber can be a serious event causing considerable disruption and economic loss. Two common approaches to solving this link loss problem are illustrated in FIG. 5 (0500).
One approach is called Line Protection Switching or 1:1 Switching (0501). This configuration (0501) consists of two point-to-point fiber pairs between two network elements (0521, 0522, 0523, 0524 and 0510). If the working fiber is lost or the signal degraded, the protection pair assumes the job of carrying the traffic between the network elements. In a fully protected system, this configuration requires four fibers (two transmit and two receive per network interface).
Another approach is called 1+1 Protective Switching (or Path Protection Switching (PPS)) in which the switching takes place at low speed or via control input to the network element (0502). With this arrangement, the traffic is sent on both the working and protective fibers. The two copies of the traffic are received at the receiving network element (0541, 0542, 0543, 0544 and 0530). Here, they are compared, and only the better copy is used.
An example of this methodology in action might configure a fiber to carry 48 channels with channels 1-24 dedicated for payload traffic and channels 25-48 used for protection. In the event one of the working channels is faulty, the receiving network element (0541, 0542, 0543, 0544 and 0530) will replace it with the other copy on the corresponding protection channel. This approach is quite fast and does not result in any loss of traffic. Problem restoration is quite efficient and the other 23 channels are not affected.
The concepts behind 1:1 and 1+1 protection have been generalized for DWDM networks to mean not just the point to point fiber pairs, but the aggregated links of the entire end-to-end lightpaths.
7. Path Protective Switching (0600, 0700, 0800)
FIGS. 6, 7 and 8 provides an example where a PSTN acts as part of the network. A central office of the PSTN can act as a node on the network. A typical path protective switching topology under normal operating conditions (0600) permits data to flow from the initial network interface (0622) through a good path to another network interface (0621), then through the central office (0610) to the telecommunications network.
Referencing FIG. 7, a typical path protective switching topology under node failure conditions (0700) permits data to flow from the initial network interface (0722) through an alternate path to another network interface (0723, 0724), then through the central office (0710) to the telecommunications network. Here since the node (0721) is down the signal is diverted to an alternate path by the upstream node (relative to the failed node (0721)).
Referencing FIG. 8, a typical path protective switching topology under link failure conditions (0800) permits data to flow from the initial network interface (0822) to a downstream node (0821) through an alternate path via network interface (0822) to another network interface (0823, 0824), then through the central office (0810) to the telecommunications network. Here since the path is down the signal is diverted by the downstream node (0821) (relative to the failed link).
In all these cases the shared protection ring can reconfigure and recover from a node or fiber failure. The switching necessary to achieve this functionality is generally implemented by multiplexer configurations similar to that illustrated in FIGS. 2 and 3 (0221, 0222, 0223, 0224, 0321, 0322, 0323, 0324).
8. Path Protection Ring Recovery Operations (0900, 1000)
Path protection switching (PPS) is generally achieved by using fields in the transmission overhead headers. In other words, the transmission specific information, i.e. destination node information, is included in each frame of data. As illustrated in FIG. 9 (0900), during normal operations of a 1+1 protection scheme, signals are placed on both fibers (0901, 0902) so that the protection fiber (0901) carries a duplicate copy of the payload, but in a different direction, and as long as the signals are received at each node on these fibers (0910, 0920, and 930), it is assumed all is well.
When a problem occurs, as illustrated in FIG. 10 (1000), such as a fiber cut between nodes B (1020) and C (1030), the network changes from a ring (loopback) network to a linear network (no loopbacks). In this example (1000), node B (1020) detects a break (1003) in the fiber, and sends an alarm to the other nodes on the working fiber (1002). The effect of the signal is to notify node C of the problem. Since node C (1030) is not receiving traffic on the protection fiber from node B (1020), it diverts its traffic onto the fiber. Node B (1020) then uses the protection fiber (1001) for this traffic Node A (1010) continues to operate as normal.
9. “Bursty” or Self-Similar Data Traffic
There is a significant difference between voice and data traffic. Voice traffic, such as telephone calls between voice network subscribers, can be very accurately modeled. This allows network planners to more easily size the capacity of a voice network infrastructure. In contrast, modern data communication traffic is far more bursty” (self-similar) than previous data traffic, in that there is more temporal self-information associated with the data than in the past. The difference between bursty and non-bursty traffic is shown in FIG. 11. Voice traffic (1102) appears bursty over very short time frames (1110). However, as the time frame increases or there is an aggregation of multiple channels of voice traffic, the data rate (1112) shown on the y-axis becomes smoother. In other words, there are fewer peaks that exceed the average aggregated data rate. However, data traffic (1104) is bursty in both short and long time intervals, and remains bursty even when aggregated with other data channels. A useful statistic in appreciating this phenomenon is that, in data networks, such as Ethernets, the peak load may often exceed the average load by a ratio of 100:1 or more. A network with statically allocated capacity that is not designed to handle the extreme peaks of the bursty traffic will not be able to throughput those peak data loads efficiently. However, designing a network that can handle even the greatest peak data rate is overly expensive and underutilized during non-peak traffic. Therefore, a need exists for a method of dynamically allocated bandwidth to handle peak data rates.
Traditional models associated with the telecommunications industry have placed both a premium and a limit on the self-information (burstiness, or ratio of peak load to average load) associated with a variety of frame relay transmission schemes. These scenarios are best summarized in the VOICE & DATA COMMUNICATIONS HANDBOOK by Regis J. Bates and Donald W. Gregory (2000, ISBN 0-07-212276-5, page 642) as follows:                “When designing a frame relay service, the speed of access is important both prior to and after installation. The customer must be aware of the need for and select a specified delivery rate. There are various ways of assigning the speed from both an access and a pricing perspective. For small locations, such as branch offices with little predictable traffic, the customer might consider the lowest possible access speed. The frame relay suppliers offer speeds that are flat rate, usage sensitive, and flat/usage sensitive combined. The flat-rate service offers the speed of service at a fixed rate of speed, whereas the usage-based service might include no flat-rate service, but a pay-as-you-go rate for all usage. The combined service is a mix of both offerings. The customer selects a certain committed information rate (CIR). The committed information rate is a guaranteed rate of throughput when using frame relay. The CIR is assigned to each of the permanent virtual circuits (PVC) selected by the user. Each PVC is assigned a CIR consistent with the average expected volume of traffic to the destination port. Because frame relay is a duplex service (data can be transmitted in each direction simultaneously), a different CIR can be assigned in each direction. This produces an asymmetrical throughput based on demand. For example, a customer in Boston might use a 64 Kbps service between Boston and San Francisco for this connection, yet for the San Francisco-to-Boston PVC a rate of 192 Kbps can be used. This allows added flexibility to meet the customer's needs for transport. However, because the nature of LANs is that of bursty traffic, the CIR can be burst over and above the fixed rate for 2 seconds at a time in some carriers' networks. This burst rate (Br) is up to the access channel rate, but many of the carriers limit the burst rate to twice the speed of the CIR. When the network is not very busy, the customer could still burst data onto the network at an even higher rate. The burst excess rate (Be) can be an additional speed of up to the channel capacity, or in some carrier's networks it can be 50 percent above the burst rate. Combining these rates, an example can be drawn as follows:        Total Throughput=CIR+Br+Be        320 Kbps total=128 Kbps+128 Kbps+64 Kbps”Thus, while the prior art permits an increase in the overall data transfer rate for short periods of time, what is not taught is any method to dramatically increase the apparent system throughput by pooling the capacity of all of the avialable communication channels, utilizing this as a resource for additional bandwidth, and allocating bursts of bandwidth to match the bursts of demand, while still maintaining protection and fallback mechanisms.        
10. Network Design and Planning
The prior art (see TELECOMMUNICATIONS ENGINEER's REFERENCE BOOK by Fraidoon Mazda (1998, ISBN 0-240-51491-2, page 22/13)) teaches that                “The first consideration when specifying any data communications network is to establish the nature and rates of traffic which the network will be expected to support both in the short and the long term. This is crucial to all network design and is the starting point of all network decisions. If errors are made here, the network cannot be expanded to meet new (and possibly unexpected) requirements.”Thus, the prior art teaches that proper planning with foresight to the future is necessary to properly design a modern communications network. However, the real problem with this philosophy in modern networks is the exponential increase in demand for bandwidth that is currently being experienced by the telecommunications industry. For example, FIG. 12 illustrates the projected growth curves (1200) being by Internet data traffic (1202). Note that the improvements in SONET capacity using time division multiplexing (TDM) (1204) is not pacing data demand. Likewise, FIG. 13 shows the projected growth rate in processing capacity (1300) of nodes in data networks. To further emphasize the exploding growth rates in data traffic, please note that the y-axis in FIGS. 12 and 13 are logarithmic. Planning in such an environment is difficult if not impossible. Couple the increased demand for bandwidth with the inherent bursty nature of the data being transmitted, and this further aggravates an already worrisome problem.        
Mazda goes on to distinguish various types of data communications traffic as follows:    18Stop-start traffic in the form of lots of short packets traveling in one direction often with slightly longer packets in the reverse direction. A characteristic of this type of traffic is that it is often associated with a requirement for very short turn-around and transit delays (e.g. word-processing). This is a classical form of asynchronous traffic.    19‘Forms’ traffic where a small amount of data travels in one direction on an ad hoc basis, but it is answered with a stream of traffic in the other direction (database enquiry, web server request).    20Block mode traffic, where there is a stream of large full packets traveling in one direction with short packets traveling in the other (file transfer).    21Transaction traffic where there are high numbers of calls with limited data transfer, often done with the Fast Select facility (e.g. credit card checks, holiday booking lounges).    22Optimized traffic, where many users are sharing a single connection (often using a Transport connection). Optimization is achieved by filling the packets as full as possible without degrading the class of service below the user requirements (OSI).    23Priority traffic. This may be any of the traffic types described above but takes precedence over the normal data flowing in the network.    24Management traffic, which is any overhead in any network.    25Multimedia traffic, such as video.    26Compressed voice traffic.
A corporate or public data network would handle all these types of traffic (and more). Most small private networks will only have one or two types of traffic and are often designed and tuned to those specific requirements. The list above is not intended to be comprehensive, but to give an idea of the differing traffic types that exist.
Service provider traffic is about 50% voice and 50% data. Studies show that data will account for as much as 96% of service provider traffic by 2005, doubling approximately yearly. With this exponential increase in network traffic loads, it is widely believed that the time has come for applying wavelength division multiplexing (WDM) not only to long-haul networks, but to metropolitan area networks (MANs), and even to access and/or campus networks. Unlike the longer-term stability seen in the traffic load in WANs (which is at least partially due to the difficult and lengthy manual set-up of expensive, leased connections), it is nearly impossible to foresee the traffic load or traffic pattern in MANs. To utilize the capacity efficiently, dynamic bandwidth on demand becomes a very important consideration in optical MANs. However, existing methods are at best web-based point-and-click (think “switchboard operator”) provisioning, and more often manual (“truck roll”) provisioning, which need at least minutes (more often days, weeks, or months) to establish lightpaths, and thus cannot meet the challenge of the bursty traffic in MANs.
Burstiness (self-similarity) is a fundamental characteristic of data traffic. In Ethernet based, data networks, indeed in Internet traffic in general, traffic has been shown over recent years to exhibit a family of related phenomena variously known as self-similarity, long-range dependency (LRD), fractal distribution, or simply burstiness. Burstiness dramatically complicates the business of designing data networks, since traditional assumptions about sizing network links no longer fit the actual data flows, and new models that work with circuit switched networks (e.g., voice networks) have not been found. This difficulty is compounded by the fact that network data flows are not symmetrical, again differing from voice traffic. Unlike voice network traffic, which becomes predictable when enough sources are aggregated together, data traffic remains bursty at all levels of aggregation, and over all timescales. The implication for network design is that data traffic cannot be adequately supported by current and proposed optical networking systems offering statically allocated, symmetrical connections. In other words, simply allocating big pipes will not allow adequate network dimensioning, since any static allocation of capacity will not adapt to “point loads” or transient traffic spikes. Further, pipes big enough to carry a bursty load some large percentage of the time will be severely underutilized most of that time. This is crucial, since data traffic, and IP traffic in particular, is expected to come to dominate networks over the next few years, with overall network traffic growth rate doubling or quadrupling every year. Since pipes big enough to handle the transient peak loads (bursts) of data traffic are empty most of the time, current optical solutions carry data on networks designed for voice; such networks exhibit extreme underutilization.
Existing metro/regional systems are overwhelmingly based on the ring topology, due to the level of familiarity and carrier comfort stemming from the long-standing adoption of SONET (an optical communications standard) rings, and due to the survivability and speed of recovery available from WDM self-healing rings (SHRs). However, OEO conversion itself can be a bottleneck, requiring very costly high-speed elements, a large equipment footprint, high energy consumption, special air conditioning, etc.
With the most advanced optical circuit-switching systems deployed today, the physical delay involved in lightpath setup and tear-down is on the order of ten milliseconds or more; network management overhead may add orders of magnitude to this delay. To support microsecond-scale lightpath setup and tear down, as well as efficient fault tolerance and cost-effectiveness, a new system for supporting high-speed WDM optical transmission is required.
Optical packet switching (OPS) would give optical networks the flexibility and granularity currently available in packet switched electronic networks. But OPS is currently a laboratory exercise, and is not practical in the short or middle term.
Proposed optical burst switching (OBS) is an intermediate solution, offering some of the advantages of both OPS and current “circuit switched” optical networks. OBS involves a one-way reservation of bandwidth (i.e., no ACK required before transmission), and optical cut-through (“switch cut-through”) of transparent data, i.e., no conversion of data to electronics at intermediate nodes. The type of bandwidth release used differentiates the various OBS approaches. Full opto-electro-optical (OEO) conversion of an out of band control channel further characterizes OBS. Since OBS network ideas are designed “from the ground up” to transmit bursts of data efficiently, OBS offers a compelling solution to some of the problems associated with bursty traffic.
One proposed OBS scheme, the “Tell and Go” prior art (Fumagalli et al., details below) has compelling advantages: no waiting for ACKs before transmitting, out of band signaling, no OEO conversion of data, a ring topology amenable to survivability, and distributed state. But there are drawbacks: the expense of transmitters and receivers on each wavelength at each node, no enforcement of fairness, the lower degree of connectivity that comes with a ring topology, and the inability to distinguish contiguous paths from the bitmask token.
The Reference Network and its extensions—the inventions disclosed here—build on ideas and architecture, now in the public domain, described in A. Fumagalli, J. Cai, I. Chlamtac, “A token based protocol for integrated packet and circuit switching in WDM rings,” published in Proceedings of Globecom 1998; which in turn compiles and incrementally moves beyond the public domain ideas of sub-carrier multiplexing (SCM: Mid-1980's) and fiber delay line optical ring architectures. (DARPA's CORD project test-bed, early 1990's). (Note, the Fumagalli scheme differs from IEEE 802.5 token ring and FDDI (Fiber Distributed Data Interface) due to its simpler station management and its simpler out-of-band traffic control, which are better suited for high-speed WDM transmission. Note specially that the IEEE 802.5 token ring cannot be used in larger rings, since, according to the standard, a station cannot generate a new token until its frame has made a complete circuit of the ring. FDDI overcomes this to some extent by allowing a node to put a new token back onto the ring once it has finished transmitting its frames. But FDDI is not suited for much more than a 100 Mbps LAN backbone, since the station management and traffic control methods, e.g., in-band source and destination addressing, synchronous frames for circuit-switched PCM or ISDN data, the three token holding timers, etc., are too complicated. While FDDI could work well at 100 Mbps, generalizing the idea to support WDM transmission at much higher speeds is improbable.)
11. Fumagalli's WDM Method
In “A TOKEN BASED PROTOCOL FOR INTEGRATED PACKET AND CIRCUIT SWITCHING IN WDM RINGS,” by A. Fumagalli, J. Cai, and I. Chlamtac, published in Proceedings of Globecom 1998, in a wavelength division multiplexed ring (1400) of W+1 channels, N nodes, and N optical links, there are W data channels, and 1 token (control/signaling) channel. The W tokens each represent a single data channel, and all tokens circulate on the token channel, which is terminated at each node. Each token consists of a channel identifier and an N bit bitmask; and advertises the availability of each link for its particular channel by a 0 to represent availability, and a 1 to represent being in use. At each node, the data channels are optically demultiplexed, and passively tapped for possible reception.
FIG. 14 illustrates a node (1400) under the Fumagalli scheme. For each of the W data channels, the node has one fixed transmitter (1402), one fixed receiver (1404), and one on/off switch (1406). For ease of illustration, only four data channels are shown here. Today, on the order of two hundred wavelength channels are multiplexed per fiber, and this number is increasing very rapidly.
The on/off switches are used to control the flow of optical signals through the node, and can prevent the circulation of “spent” packets in the ring. The optical delay line (1408) at each node is used to delay the data transmission, giving the node enough time to process the control packet (token). The information on the control channel (1410) is handled by a controller (1412). A buffer of sufficient size is provided at each node for data processing (1414) to queue incoming (internetwork) transmissions prior to their transmission into the ring (intra-ring) and outgoing (extranetwork) transmissions prior to their transmission out of the ring. Though this design is somewhat similar to IEEE 802.5 token ring and FDDI (Fiber Distributed Data Interface), the simpler station management and, especially, the simpler out-of-band traffic control are much more suitable for high-speed WDM transmission.
Using this scheme, a node that has data to transmit simply claims the available communication path, sets the appropriate fields in the token (if a link is to be claimed for a communication path, the bit corresponding to that link is set to 1 to reflect this), and releases the token and the data for transmission downstream simultaneously. The node does not need to check with the other nodes or any central authority. This is known in the literature as a “tell-and-go” (TAG) scheme. The destination node receives the token on the token (control) channel while the data is delayed on the data delay loop (1408), monitors the receiver that taps that channel to receive the data, and opens the ON/OFF switch (1406) on that channel to prevent interference downstream, and thereby “clean up” that data channel. Meanwhile, upon completion of its transmission, the source node waits for the token to return and then regenerates the token, clearing the fields corresponding to its communication path. This protocol is fully distributed, i.e., there is no central network controller.